Method and apparatus for construction machine visualization

ABSTRACT

A system for displaying information to an operator of a machine comprises a head tracking system and a projection system. One system for displaying information uses a projection system and a see-through display to present two-dimensional images to an operator. One system uses a projection system and a see-through display to present three-dimensional images to an operator. One system uses a pair of smart glasses to display information to a user based a direction a user is looking.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/789,493 filed Jul. 1, 2015, which claims the benefit of U.S.Provisional Patent Application No. 62/020,521 filed Jul. 3, 2014, allare incorporated herein by reference in their entirety.

BACKGROUND

The present disclosure relates generally to the display of information,and more particularly to construction machine visualization.

Operators of construction machines are required to operate the machinesaccording to a desired site plan in order to modify the site to matchthe desired site plan. The operator is required to continuously monitorthe current state of the site and operate the machine in a manner tochange the current site to the desired site plan. Typically, an operatorof a construction machine must constantly reference a desired site plan,typically on paper or on a display in the cab of the machine, to ensurethat the modification of the current site is performed to change thesite to the desired state.

SUMMARY

In one embodiment, a method includes determining a direction of a user'sgaze. The direction of the user's gaze can be determined using a headtracking system. The direction of the user's gaze can also be determinedusing a gaze tracking system. Information to be displayed to a user isthen determined and the information is displayed to the user. Theinformation can be displayed by transmitting the information to aprojection system and/or a pair of smart glasses. In one embodiment,changes in the user's gaze and changes in the current state of aconstruction site are determined and new information is generated anddisplayed to the user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a construction machine, specifically, an excavator;

FIG. 2 depicts an operator's view from inside a cab of a constructionmachine according to an embodiment;

FIG. 3 depicts an operator's view from inside a cab of a constructionmachine according to an alternative embodiment;

FIG. 4 depicts a pair of smart glasses according to an embodiment;

FIG. 5 depicts a system for displaying information to a user via atwo-dimensional projection system according to an embodiment;

FIG. 6 depicts a system for displaying information to a user via athree-dimensional projection system according to an embodiment;

FIG. 7 depicts a system for displaying information to a user via a pairof smart glasses according to an embodiment;

FIG. 8 depicts an example of a display according to an embodiment;

FIG. 9 depicts a flowchart of a method for displaying informationaccording to an embodiment;

FIG. 10 depicts a flowchart of a method for displaying informationaccording to an alternative embodiment; and

FIG. 11 depicts a high-level schematic of a computer according to anembodiment.

DETAILED DESCRIPTION

FIG. 1 depicts excavator 100 comprising cab 102, boom 104, stick 106,and bucket 108. Cab 102, boom 104, stick 106, and bucket 108 arerotatably mounted on an undercarriage 110. An operator sits in cab 102and controls excavator 100 to move it to a desired location andmanipulate various material, such as dirt. The accuracy and efficiencyof operations facilitated by an operator vary depending on theexperience of the operator. Accurate and efficient operation ofexcavator 100 typically requires an operator with extensive experienceincluding interpreting site plans and modifying a site using aconstruction machine to achieve the desired site plan.

FIG. 2 depicts an operator's view from inside cab 102 of excavator 100according to one embodiment of the invention. From inside cab 102, anoperator can see stick 106 and bucket 108. In addition, according to oneembodiment, the view from inside cab 102 includes information overlay200 (also referred to as a heads up display or HUD) which, in thisexample, depicts an image of excavator 100 and a desired plan to modifythe earth around excavator 100. As shown in FIG. 2, an operator can seethe view of a site outside cab 102 as well as information overlay 200.An operator can select and/or change information displayed byinformation overlay 200 via intelligent display 202.

FIG. 3 depicts an operator's view from inside cab 102 of excavator 100according to one embodiment. Similar to FIG. 2, from inside cab 102, anoperator can see boom 104, stick 106 and bucket 108. As operator isoperating excavator 100, additional information is displayed includingexcavation area overlay 302 which depicts an area which is to bemodified according to a predetermined desired site plan. Volume overlay304 provides an operator with information concerning a volume of earthto be removed from excavation area 302. Completion overlay 306 providesan operator with an indication of the level of completion of anoperation. In one embodiment, additional overlays are presented to theoperator. For example, an overlay pertaining to a current position ofbucket 108 compared to an expected position of bucket 108 can bedisplayed. In addition, information about excavator 100 engine rotationsper minute (RPM), instantaneous fuel rate, fuel consumption, and otherinformation received from excavator's electronic control unit (ECU) viavarious protocols can be displayed. In one embodiment, protocols can becontrol area network (CAN) bus protocols such as CAN J1939, CAN open, orCAN custom.

Information may be displayed to an operator using various hardware andmethods. In one embodiment, information is displayed to an operator viaa pair of smart glasses which incorporate display devices. FIG. 4depicts smart glasses 400 including display devices which are used toprovide a wearer with information by displaying information over thewearer's view through the smart glasses. Glasses 400 are used in oneembodiment to provide information overlay 200 of FIG. 2 and excavationarea overlay 302, volume overlay 304, and completion overlay 306 of FIG.3. In another embodiment, a projection system is used to displayinformation to an operator by projecting an image onto a transparentsurface such as a windshield of a machine.

FIG. 5 depicts system 500 for advanced visualization of site informationusing overlays. Controller 502, located in a cab of a machine in oneembodiment, manages the system and receives head tracking informationvia head tracking driver 504 from head tracking system 506 (also locatedin the cab of the machine in one embodiment). Controller 502 is also incommunication with projection system 508 via projection driver 510.Projection system 508 comprises see-through display 522 and projectionsource 524. Controller 502 is also in communication with intelligentdisplay 512. Specifically, software 514 of controller 502 is configuredto allow controller 502 to communicate with screen 516 of intelligentdisplay 512. Touch driver 518 of controller 502 is in communication withtouch sensitive component 520 of intelligent display 512 and also withsoftware 514. Intelligent display 512 can be used by an operator toselect and/or change information displayed to a user via see-throughdisplay 522 and projection source 524. Each of these components aretypically located in a cab of a machine but can be located or attachedto other areas of the machine.

In one embodiment, controller 502 receives information from headtracking system 506 via head tracking driver 504 in order to determinewhere an operator is looking based on the operator's head position.Based on where the operator is looking, controller 502 determines whatand where information should be presented to a user via an overlay (orHUD) using projection system 508. Information for display via projectionsystem 508 can be selected and/or changed via intelligent display 512using touch sensitive component 520.

In one embodiment, projection system 508 is a monoscopic systemcomprising a transparent surface and a light source. The transparentsurface is used to visualize an image generated by the light source. Thesurface can cover or overlap a portion of a windshield or the entirewindshield of a vehicle such as excavator 100 (shown in FIG. 1). Thelight source, in one embodiment, is capable of creating an appropriatelight wave for the corresponding transparent surface in order to displayinformation to the operator of excavator 100 (shown in FIG. 1). An imagegenerated by the light source and transparent surface can be focused ata desired distance or at an infinite distance.

In one embodiment, head tracking system 506 is capable of evaluating aposition of an operator within cab 102 (shown in FIG. 1). Head trackingsystem 506, in one embodiment, provides information about the currenthead position of the operator to controller 502 via head tracking driver504. Images displayed to the operator are then varied based oninformation from head tracking system 506. For example, the contentvisualized on the display or location of the information on the displaymay change based on the operator's head movement and current position.

Controller 502, in one embodiment, stores and manages digital designinformation such as surfaces, line work, and other site data. Controller502 manages machine geometry calculations such as excavation andnavigation calculations. In one embodiment, controller 502 managescommunications with a site communications system and other localmachines (not shown). Controller 502, in one embodiment generates imagesfor display via one or more display systems such as projection or smartglasses. Controller 502 can determine a current state of a site usinginformation from various sensors attached to the machine such as imagesensors, object sensors, etc. Information from these sensors is receivedby controller 502 which, in one embodiment, uses the information togenerate a virtual representation of the current state of the site. Thisvirtual representation can be used to determine differences between thecurrent state of the site and a desired site plan.

Intelligent display 512, in one embodiment, is a high-brightness,multi-touch display device and is the primary device for user input andselection. In one embodiment, intelligent display 512 displays a twodimensional image of a machine in which intelligent display 512 islocated and provides machine and site data to an operator. Intelligentdisplay 512 can display a machine dashboard and provide informationconcerning the machine's current condition as well as provide machinediagnostics. Machine parameters can be received by controller 502 andtransmitted to a device, such as intelligent display 512, for display tothe user.

As described above, system 500 can be used to provide an operator withvarious information including site and machine operation via a heads-updisplay as well as machine conditions and diagnostics via intelligentdisplay 512. System 500 displays two-dimensional information using a HUDvia projection system 508. In other embodiments, the display ofinformation to a user can be a three-dimensional or a display generatedusing smart glasses.

FIG. 6 depicts system 600 for displaying information viathree-dimensional images. Controller 602 functions similar to controller502 shown in FIG. 5 and includes head/eye/gaze tracking driver 604 forreceiving information from head/eye/gaze tracking system 606. In orderto generate three-dimensional images that are viewable by an operator,system 600 determines an operator's gaze by tracking the movement of theoperator's eyes. The gaze and eye information, along with head positioninformation, is used by system 600 to dynamically manage the parallaxfor the operator's eyes. Projection system 608 comprises stereo readysee-through display 622 onto which three dimensional images areprojected by stereo projection source 624. Display information forprojection system 608 is transmitted, in one embodiment, stereoprojection driver 610 of controller 602 to stereo projection source 624of projection system 608. Intelligent display 612 comprising screen 616and touch sensitive component 620 operate in a manner similar tointelligent display 512 of FIG. 5.

In one embodiment, information is displayed to an operator via a pair ofsmart glasses such as smart glasses 400 shown in FIG. 4. FIG. 7 depictsa system in which information is displayed to an operator via a pair ofsmart glasses. Controller 702 functions similar to controller 502 shownin FIG. 5 and includes a head/eye/gaze tracking driver for receivinginformation from head/eye/gaze tracking system 706. In order to generatethree-dimensional images that are viewable by an operator, system 700determines an operator's gaze by tracking the movement of the operator'seyes. This gaze and eye information, along with head positioninformation, is used by system 600 to dynamically manage the parallaxfor the operator's eyes. In addition, head, eye, and gaze tracking canbe used to generate a 360 degree visualization of virtual informationaround the operator. In contrast with projection systems which projectinformation onto a fixed transparent display (e.g. a windshield of avehicle), system 700 allows information to be displayed to a user in anydirection the user looks since the display of information is presentedto the user via display devices incorporated in the associated pair ofsmart glasses 708. Smart glasses 708, in one embodiment, comprisesstereo ready see-through display 722 onto which three dimensional imagesare projected by stereo projection source 724. Display information forsmart glasses 708 is transmitted, in one embodiment, from wirelesscommunication port 726 of controller 702 to stereo projection wirelesscommunication port 728 of smart glasses 708. Information received viawireless communication port 728 is sent to stereo projection source 724of smart glasses 708. Images are then displayed to the operator viasmart glasses 708. Intelligent display 712 comprising screen 716 andtouch sensitive component 720 operates in a manner similar tointelligent display 512 of FIG. 5.

FIG. 8 depicts an example of a display according to one embodiment usingprojector 802 and display surface 804. Projector 802 projects an imageonto display surface 804. Similarly, projector 806 projects an imageonto display surface 808. In various embodiments, different projectorsand display surfaces are used. For example, a laser scan projector (scanline) can be used with a reflective film to scatter light emitted fromthe projector. A laser matrix projector can be used with a reflectivefilm to scatter light emitted from the projector. A laser scan projector(full scan) can be used with reflective film to scatter light emittedfrom the projector. In one embodiment, a projector is used with areflective film sized to be used as a display in conjunction with awindow and/or view port of a cab of a machine. In one embodiment, aprojector is used with a reflective film sized to be used with a pair ofsmart glasses. Smart glasses can be monocular (i.e., displayinginformation via one lens of the smart glasses), or binocular (i.e.,displaying information via both lenses of the smart glasses). In oneembodiment, a three dimensional image can be displayed to the user. Inother embodiments, a mix of two dimensional and three dimensional imagesare displayed to the user.

FIG. 9 depicts a flow chart of a method 900 for displaying informationto a user according to one embodiment. Various configurations asdescribed above can operate using the steps identified in method 900. Atstep 902, sensor input from various sensors (previously described) arereceived and analyzed. At step 904, a current head position of a userand/or operator is determined. At step 906 the current head position anda previous head position are compared. At step 908, relative headmovement is determined in order to evaluate whether displayedinformation should be changed based on head movement. At step 910,information is transmitted to the display system based on possiblechange in head position as well as changes in conditions monitored(e.g., movement of the machine or parts of the machine, etc.)

FIG. 10 depicts a flow chart of a method 1000 for displaying informationto a user of a machine according to an embodiment. At step 1002 adirection of a user's gaze is determined. In one embodiment, a user'sgaze is determined based on a head position of the user. For example,controller 502 (shown in FIG. 5) receives information from head trackingsystem 506 via head tracking driver 504 in order to determine where auser is looking based on the user's head position. In one embodiment, auser's gaze is determined based on a position of the user's eyes and adirection the user's eyes are looking. For example, controller 602(shown in FIG. 6) receives information concerning a user's gaze fromhead/eye/gaze tracking system 606 via head/eye/gaze tracking driver 604.

At step 1004, information to be displayed to the user is determinedbased on the direction of the user's gaze and a current state of a site.For example, the information to be displayed to the user can include howa portion of a site the user is looking at needs to be modified. In oneembodiment, the information to be displayed to the user is determined bydetermining a difference between the current state of the site and adesired site plan. An operation to be performed based on the differentbetween the current state of the site and the desired site plan can alsobe determined. The information displayed to the user can include theoperation to be performed. In one embodiment, the difference between thecurrent state of the site and the desired site plan is used to generatea value pertaining to a percent of completion of the site. The valuepertaining to the percent of completion of the site can be displayed tothe user.

At step 1006, the information is displayed to the user. In oneembodiment, the information is displayed by transmitting the informationfrom controller 502 (or 602) to a projection system, such as projectionsystem 508. In one embodiment, the information is displayed bytransmitting the information from controller 502 (or 602) to a pair ofsmart glasses, such as smart glasses 400 (shown in FIG. 4).

At step 1008, it is determined that a user's gaze has changed. A changein the user's gaze, in one embodiment, is determined using informationfrom head/eye/gaze tracking system 606. In another embodiment, a changein the user's gaze is determined using information from head trackingsystem 506.

At step 1010, it is determined that the current state of the site haschanged. In one embodiment, controller 502 (or 602) determines that thecurrent state of the site has changed using input received from varioussensors associated with and/or attached to a machine the user isoperating.

At step 1012, new information to be displayed to the user is determined.This new information can be based on a change in the user's gaze and/ora change in the current state of the site. At step 1014, the newinformation is displayed to the user.

Smart glasses 400, 708, intelligent display 202 and the variouscomponents of systems 500, 600, and 700 can each be implemented using acomputer. A high-level block diagram of such a computer is illustratedin FIG. 11. Computer 1102 contains a processor 1104 which controls theoverall operation of the computer 1102 by executing computer programinstructions which define such operation. The computer programinstructions may be stored in a storage device 1112, or other computerreadable medium (e.g., magnetic disk, CD ROM, etc.), and loaded intomemory 1110 when execution of the computer program instructions isdesired. Thus, the method steps of FIGS. 9 and 10 can be defined by thecomputer program instructions stored in the memory 1110 and/or storage1112 and controlled by the processor 1104 executing the computer programinstructions. For example, the computer program instructions can beimplemented as computer executable code programmed by one skilled in theart to perform an algorithm defined by the method steps of FIGS. 9 and10. Accordingly, by executing the computer program instructions, theprocessor 1104 executes an algorithm defined by the method steps ofFIGS. 9 and 10. The computer 1102 also includes one or more networkinterfaces 1106 for communicating with other devices via a network. Thecomputer 1102 also includes input/output devices 1108 that enable userinteraction with the computer 1102 (e.g., display, keyboard, mouse,speakers, buttons, etc.) One skilled in the art will recognize that animplementation of an actual computer could contain other components aswell, and that FIG. 11 is a high level representation of some of thecomponents of such a computer for illustrative purposes.

The foregoing Detailed Description is to be understood as being in everyrespect illustrative and exemplary, but not restrictive, and the scopeof the inventive concept disclosed herein is not to be determined fromthe Detailed Description, but rather from the claims as interpretedaccording to the full breadth permitted by the patent laws. It is to beunderstood that the embodiments shown and described herein are onlyillustrative of the principles of the inventive concept and that variousmodifications may be implemented by those skilled in the art withoutdeparting from the scope and spirit of the inventive concept. Thoseskilled in the art could implement various other feature combinationswithout departing from the scope and spirit of the inventive concept.

1. A method for displaying information to an operator of a constructionmachine, the method comprising: determining a direction of a gaze of theoperator located in a cab of the construction machine; generating animage to be displayed to the operator based on the direction of the gazeof the operator, a representation of a construction site, a desired siteplan, and a construction machine operation to be performed, the imagecomprising an excavation area overlay, a volume overlay, and acompletion overlay; and displaying the image to the operator.
 2. Themethod of claim 1, further comprising: determining a difference betweenthe representation of the construction site and the desired site plan;and determining the construction machine operation to be performed basedon the difference.
 3. The method of claim 1, wherein the constructionmachine operation to be performed comprises movement of an implement ofa construction machine based on a current position of the implement anda desired position of the implement.
 4. The method of claim 1, whereinthe direction of the gaze of the operator is determined based on a headposition of the operator.
 5. The method of claim 1, wherein thedirection of the gaze of the operator is determined based on a positionof the operator's eyes and a direction the operator's eyes are looking.6. The method of claim 1, wherein the generating the image to bedisplayed to the operator comprises: determining a difference betweenthe representation of the construction site and the desired site plan;and generating a value pertaining to a percent of completion of theconstruction site based on the determining the difference between therepresentation of the construction site and the desired site plan,wherein the completion overlay comprises the percent of completion ofthe construction site.
 7. The method of claim 1, further comprising:determining that the gaze of the operator has changed; determining a newimage to be displayed to the operator based on the determining that thegaze of the operator has changed; and displaying the new image to theoperator.
 8. The method of claim 1, wherein the image displayed to theoperator further comprises an overlay pertaining to a current positionof an implement and a desired position of the implement.
 9. An apparatusfor displaying information to an operator of a construction machine, theapparatus comprising: a processor; and a memory to store computerprogram instructions, the computer program instructions when executed onthe processor, cause the processor to perform operations comprising:determining a direction of a gaze of the operator located in a cab ofthe construction machine; generating an image to be displayed to theoperator based on the direction of the gaze of the operator, arepresentation of a construction site, a desired site plan, and aconstruction machine operation to be performed, the image comprising anexcavation area overlay, a volume overlay, and a completion overlay; anddisplaying the image to the operator.
 10. The apparatus of claim 9, theoperations further comprising: determining a difference between therepresentation of the construction site and the desired site plan; anddetermining the construction machine operation to be performed based onthe difference.
 11. The apparatus of claim 9, wherein the constructionmachine operation to be performed comprises movement of an implement ofa construction machine based on a current position of the implement anda desired position of the implement.
 12. The apparatus of claim 9,wherein the direction of the gaze of the operator is determined based ona head position of the operator.
 13. The apparatus of claim 9, whereinthe direction of the gaze of the operator is determined based on aposition of the operator's eyes and a direction the operator's eyes arelooking.
 14. The apparatus of claim 9, wherein the generating the imageto be displayed to the operator comprises: determining a differencebetween the representation of the construction site and the desired siteplan; and generating a value pertaining to a percent of completion ofthe construction site based on the determining the difference betweenthe representation of the construction site and the desired site plan,wherein the completion overlay comprises the percent of completion ofthe construction site.
 15. The apparatus of claim 9, the operationsfurther comprising: determining that the gaze of the operator haschanged; determining a new image to be displayed to the operator basedon the determining that the gaze of the operator has changed; anddisplaying the new image to the operator.
 16. A computer readable mediumstoring computer program instructions for displaying information to anoperator of a construction machine, which, when executed on a processor,cause the processor to perform operations comprising: determining adirection of a gaze of the operator located in a cab of the constructionmachine; generating an image to be displayed to the operator based onthe direction of the gaze of the operator, a representation of aconstruction site, a desired site plan, and a construction machineoperation to be performed, the image comprising an excavation areaoverlay, a volume overlay, and a completion overlay; and displaying theimage to the operator.
 17. The computer readable medium of claim 16, theoperations further comprising: determining a difference between therepresentation of the construction site and the desired site plan; anddetermining the construction machine operation to be performed based onthe difference.
 18. The computer readable medium of claim 16, whereinthe construction machine operation to be performed comprises movement ofan implement of a construction machine based on a current position ofthe implement and a desired position of the implement.
 19. The computerreadable medium of claim 16, wherein the direction of the gaze of theoperator is determined based on a head position of the operator.
 20. Thecomputer readable medium of claim 16, wherein the direction of the gazeof the operator is determined based on a position of the operator's eyesand a direction the operator's eyes are looking.